Sensible Scenes : Visual Understanding of Complex

Visual Understanding of Complex Structures through Causal Analysis Matthew Brand, Lawrence Birnbaum, and Paul Cooper Northwestern University The Institute for the Learning Sciences 1890 Maple Avenue, Evanston IL 60201 [email protected] Abstract An important result of visual understanding is an explanation of a scene's causal structure: How action|usually motion|is originated, constrained, and prevented, and how this determines what will happen in the immediate future. To be useful for a purposeful agent, these explanations must also capture the scene in terms of the functional properties of its objects|their purposes, uses, and a ordances for manipulation. Design knowledge describes how the world is organized to suit these functions, and causal knowledge describes how these arrangements work. We have been exploring the hypothesis that vision is an explanatory process in which causal and functional reasoning plays an intimate role in mediating the activity of low-level visual processes. In particular, we have explored two of the consequences of this view for the construction of purposeful vision systems: Causal and design knowledge can be used to 1) drive focus of attention, and 2) choose between ambiguous image interpretations. Both principles are at work in SPROCKET, a system which visually explores simple machines, integrating diverse visual clues into an explanation of a machine's design and function. Visual understanding A fundamental purpose of vision is to relate a scene to the viewer's beliefs about how the world ought to be|to \make sense" of the scene. Understanding is the preparation we make for acting, hence our beliefs are fundamentally causal in nature; they describe the world's capacity for action and change. \Making sense" of a scene means assessing its potential for action, whether instigated by the agent, or set in motion by forces already present in the world. This work was supported in part by the National Science Foundation, under grant number IRI9110482. The Institute for the Learning Sciences was established in 1989 with the support of Andersen Consulting, part of The Arthur Andersen Worldwide Organization. The Institute receives additional support from Ameritech, North West Water Plc, Institute Partners, and from IBM. G1 G3 A1 A2